-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Setup CI action to automate testing #103
base: main
Are you sure you want to change the base?
Conversation
This comment was marked as resolved.
This comment was marked as resolved.
Co-authored-by: Jim Huang <[email protected]>
It's a good chance to also include building these projects as regression testing https://github.com/fuhsnn/widcc/?tab=readme-ov-file#building-real-world-projects And possibly with ASAN on, I'll test locally first. I'm new to Github actions, but this action seems to worth using, any pro and cons of caching in your experience? |
I haven't done any package caching in actions before, but current speed might be fast enough as long as matrix count is low and we reuse instances as much as possible. Unless there are some critical scenarios where jobs must be separate out. |
I'm leaning towards running Dockerfiles with Github actions, I like having a more controllable environment and the ability to easily test multi-arch builds with https://github.com/docker/setup-qemu-action is nice. |
Running SQLite and curl through automated GitHub Actions workflows makes a lot of sense. This approach helps catch both regressions and compatibility issues. We can also add continuous performance monitoring, similar to rv32emu's approach. The benchmark results automatically appear as comments in pull requests, making it easy for both contributors and reviewers to see how code changes impact performance compared to the default branch. Another example: automated tests of ThreadX |
I am using nektos/act to test GitHub Actions locally, unfortunately, while it's all passed on GitHub Actions, on local Docker it will fail on test 1542 (after installing impacket, test 1451 also fail). Currently I am looking on curl's official workflow to see if there's any hint on these 2 failures. |
That's why I prefer Docker for running the builds, you see, these code bases are decades old, some tests expect old features or behaviors from system tools that are deprecated or slightly changed over updates; a test failing don't always mean that the compiler is faulty. With Docker I can control exactly which dependency to install at what version on a per-project basis, which makes sense for regression testing (that is, what previously worked should keep working in the same non-changing environment). I'm working on Docker based test jobs, just leave it to me. |
Relates to #101, this PR adds a CI action file for testing test suites and self-hosting integrity when user pushes commits or opens a pull request, to prevent source code being incorrect on committed.
I haven't added an action for code style checking since there's no code style formatting guideline in the repository.